Spatial localization using augmented reality

ABSTRACT

Methods and system for locating an event on an aircraft using augmented-reality content, an array of ultrasonic devices configured in mesh topology, and deep learning.

CROSS-REFERENCE TO RELATED APPLICATIONS

None.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

None.

FIELD

The non-limiting technology herein relates to use of an ultrasoundsensors network (herein called “Ultrasound”) to automatically detect theuser location coordinates in an indoor environment with AugmentedReality (herein called “AR”) including mobile devices, that enables theuser to receive location information in real time; and the use of DeepLearning (herein called “DL”) methods with the purpose of improving theaircraft maintenance process, guiding technicians to locate the elementsthat need to be accessed during the maintenance quickly, with accuracyand precision. In some other aspects, the technology herein relates tomethods and systems for locating and navigating an environment, usingultrasound, in some embodiments in combination with deeplearning/artificial intelligence image pattern recognition, to assist auser using augmented-reality content on a display device.

BACKGROUND

Currently, the aeronautical maintenance business is facing difficultiesin obtaining maintenance technicians to perform their required tasks. Inmany cases, technicians have inadequate knowledge about the aircraftsthat need to be serviced; this leads to the maintenance task being morecostly as it may take longer or requiring more than one mechanic toperform the task.

In order to carry out the maintenance tasks, the mechanic typically mustinitially check if there is any available information about the aircraftin which some maintenance action is required. If there is no advancedinformation about the need for maintenance on the aircraft, the mechanicshould check the logbook of the aircraft to see if there is any pendingmaintenance to be performed. For the situation in which there is aproblem with the aircraft, the first approach is to try to correct theproblem. In the case of problems that are not simple, it is possible toverify dispatch of the activity through the use of the Minimum EquipmentList (MEL), which is used for determining the aircraft's continuedairworthiness. If the defective equipment does not appear in the MEL, itis necessary to perform troubleshooting, that is, to try to solve theproblem through a systematic search for the root cause of the problem ofthe component or its replacement.

This troubleshooting procedure must typically be carried out until theproblem can be solved and the aircraft is operational again. If there isa problem in the aircraft, in which the mechanic must performmaintenance procedures, it is necessary to know the location ofparticular components in the aircraft that may have failed or needsattention, in addition to obtaining the necessary parts, equipment andmaterials to carry out this activity. The necessary maintenanceinformation can be found in an aircraft maintenance manual (maintenancemanuals may contain, for example, removal/installation procedures,troubleshooting, activation/deactivation, etc.) and can be accessed bythe mechanic via a printed document and/or a digital file. The mechanicmust typically follow all the maintenance steps outlined in the aircraftmaintenance manual to ensure proper maintenance of the equipment.

From the aforementioned, the need to have prior knowledge about theaircraft becomes apparent in order to quickly locate and maintain thecomponent that will need the maintenance. In many cases, the componentsthat need to be located are not easily found as they may be hiddenwithin, by or behind, other components of the aircraft such as e.g.,panel covers. The faster the maintenance location is found, the fasterthe maintenance activities can be started.

The difficulty of locating a component can be reduced by coupling ageolocation system between the equipment and the mechanic. Geolocationis used for many different purposes, for example: for navigating inunknown environments, locating objects and identifying places, amongother things. However, there is no technology suitable for aircraftmaintenance that presents high positioning accuracy, simultaneously, inindoor and outdoor environments. That is, there is no solution that isreliable, effective and efficient for such purposes.

Currently there are a lot of technologies that allow localization. Mostof the time, the solution used is based on GPS systems which provideabsolute position geocoordinates at high precision. However, GPS haslimitations when used indoors or beneath obstructions such as in anairplane cargo bay or under an aircraft fuselage. For GPS to work well,the receiver needs to be within line of sight of GPS satellites. Thus,to overcome the deficiencies of GPS systems, in the last decades,several approaches have emerged that propose solutions, of hardware andsoftware, with adaptations for locations in closed environments.Meanwhile, high location accuracy is helpful for correctly finding theitem being searched for and not direct the maintenance user to the wronglocation to not delay the execution of a maintenance task.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description of exemplary non-limitingillustrative embodiments is to be read in conjunction with the drawingsof which:

FIG. 1 illustrates example non-limiting components (hardware, software,server) in a prior art AR-Based System.

FIG. 2 shows a non-limiting embodiment of an example non-limiting basicAR processing flow.

FIGS. 3A, 3B and 3C show non-limiting examples of how events arecaptured and how they are later used in the non-limiting technology.

FIG. 4 shows an example non-limiting flowchart of the methodology usedin FIGS. 3A, 3B & 3C.

FIGS. 5A & 5B are non-limiting examples wherein the non-limitingtechnology is applied.

FIG. 6 illustrates a non-limiting embodiment of an array of sensors.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Example non-limiting embodiments find application in improving anaircraft's maintenance process, and quickly guiding the technicians tolocate the elements that need to be accessed during the maintenance,with accuracy and precision.

The technology proposed is suited to provide maintenance for a varietyof environments and vehicles not limited to: indoor environment, complexstructures, aircraft, automobiles, and watercrafts. More specifically,the technology's preferred embodiments in one potential application areincorporated into an aircraft's maintenance system.

Ultrasound technology can be used, among other applications, to locateusers in an environment with great precision. As is well known, theoperating principle is based on the response time of propagation ofultra-high frequency sound waves emitted by transceiver devices. Sensorsmeasure time of flight or time of arrival (absolute or differential) ofemitted ultrasonic pulses. Distance can be calculated for each pathbetween an emitter(s) and the sensor(s) based on the speed of sound. Theuser's position can be obtained by trilateration of signals fromtransceiver devices installed in the environment and on the user or on adevice the user carries. While such ultrasound technology has been usedin the past to determine the pose of a body part of a user (e.g., ahand) relative to an array of sensors and/or emitters (see the NintendoPower Glove manufactured by Mattel as one example), the presentnon-limiting technology extends such approaches to make it useful inaircraft maintenance contexts.

Augmented Reality (“AR”) is a real-time representation combining thereal, physical world and the virtual world within a common userinterface. Generally, the virtual information is rendered from the real,physical world using an augmented reality device which may constitute adisplay device such as a handheld display, goggles or glasses. One ARapproach employs transparent lenses to allow the user see the real worldthrough the licenses while displaying virtual objects within the realworld scene. Another AR approach uses a camera to capture the real worldscene, and combines the captured camera video stream with a virtualartificial display object(s) that appears to the user to be part of thereal world scene. Still another AR approach models the real world using3D graphics techniques, and displays scenes defined by a model of theworld plus an additional virtual object(s) that is not part of the world(this approach is sometimes referred to as “mixed reality”).

There are two interrelated elements in most AR-Based Systems. The firstelement comprises the hardware and software components utilized toimplement the AR-Based System. The second element comprises ways inwhich AR is implemented in a real environment. Three components used inmany AR-Based Systems include: the Hardware (100), the Software (200),and a Server (300). FIG. 1 explains the relationship between the threecomponents and how they make AR-Based Systems work.

The hardware (100) in an AR device typically uses modules like DisplayDevices (102) (e.g., head mounted display (HMD), smartphone screen,eyeglasses, etc.), Input Devices (104) (e.g., sensors such as a: camera,infrared sensor, depth sensor, GPS, gyroscope, accelerometer, etc.), anda Processor (106). The sensors (105) detect the position and theorientation of the display device (102) in or relative to anenvironment.

The software (110) aspect of an AR device is used in rendering andinjecting virtual images into the real world. Software (110) (e.g.,StudioMax, Cinema4D, AutoCAD 3D, etc.) is often tasked with generatingthe virtual images used for overlapping over or combining with liveimages.

As an AR device requests certain virtual images, the server (120) (e.g.,web, cloud, etc.) retrieves and sends the virtual images to the ARdevices and is also typically capable of storing virtual images forlater use. Some AR systems do not use a server (120) but insteadgenerate virtual images locally or with a peer-to-peer or other computeror network architecture.

The way an AR device interacts with the real world often depends on theavailability of the environment. Marker-based and Location-based ARinterfaces are two main means of associating the real world with thevirtual world. Marker-based AR often has prior knowledge about theenvironment that the device “sees,” whereas location-based AR often doesnot. Location-based AR works by locating a reference point (e.g.absolute or relative position) in the environment the user is in.

In one example non-limiting implementation, the basic AR processing flow(200) begins with an image captured by a camera or CMOS image sensor(202). This video is broken into frames (204). Each image/frame isprocessed (206) to detect a marker (220), which in turn is used todetermine to identify a position or pose, e.g., relative to a marker.When this marker is detected, the camera position (and in some casesalso its orientation) is calculated considering its intrinsic parameterse.g., relative to the marker and thus the environment. Once thisposition/orientation/pose is defined, the virtual object(s) (208) isrendered in the same image/frame and the translation, rotation, andperspective angles for virtual content are applied for display (210).Such technology is used as one example in the context of video gaming,see “AR Cards” and associated gaming applications of the Nintendo 3DShandheld 3D video game system.

The fiducial marker (220) is a technique that is often used to allow theAR system to accurately position a virtual object in the real, physicalworld. The fiducial marker (220) is typically a bidimensional ormultidimensional object that is positioned in the scene to be captured,detected by the camera, and then processed, to identify the object'sposition, as exemplified in FIG. 2. In one embodiment, the fiducialmarker (220) comprises a sticker or card that bears a special patternwhich is (a) easily recognizable by an image decoder or patternrecognizer, (b) is distinguishable from other fiducial markers alsoplaced in the scene (e.g., it is encoded with a unique,optically-recognizable identifier), and (c) in some cases may bemeasurable to allow an optical detector to infer its pose (e.g.,position and/or orientation or aspects thereof) based on the opticallydetected pattern. Other fiducial marks may comprise infrared patternsand/or emitters, beacons that emit energy, or other arrangements asknown in the art.

It is also possible to use the AR without any artificial fiducialmarkers or other such elements (220). In this case, a device such as acamera can capture position, orientation and/or pose by detectingnatural features in the real, physical world. One example of thistechnique is the identification by edges or textures of an object basedon characteristics of a corresponding 3D model. The correspondence ofedges and/or textures permits the natural object to itself serve as themarker, without the need to use a dedicated artificial fiducial markingobject (220) that is not part of the real, natural physical world.

Much advancement in this technology is making it possible to use the ARin the aeronautical industry. The 3D models created during aircraftdesign can be reused, allowing the use of AR in aircraft manufacturing,training, inspection, and maintenance. The use of an AR-based deviceprovides accessibility by showing virtual information in the real,physical world.

The first step of AR tracking is the detection of a known target in anincoming video stream using a detection algorithm, yielding the “pose”(e.g., position and orientation in 6 degrees of freedom) of the camerarelative to the target. The detection procedure involves finding a setof matches between the incoming images and a reference image(s), butrobust and apt detection of objects for AR is still a challengingproblem. Deep Learning techniques will be used to address these targetdetection problems, since deep convolutional neural networks can betrained to detect targets for augmented reality tracking. The targetimage is rendered to create many synthetic views from different anglesand under different illumination conditions. Therefore, apart fromspeeding up the classification of the quality or state of the componentsof the aircraft in the process of identifying faults and defects, deeplearning allows these processes to be performed by technicians with alow level of specialization, making them cheaper and allowing smallernumbers of interventions.

The non-limiting technology described herein relates to systems andmethods for spatial location of three-dimensional points usingcombinations of augmented reality, ultrasound (or other geolocationsystem by active sensors), computer vision and Deep Learning. Theexemplary system uses environment reconstruction techniques togetherwith ultrasound to realize the spatial location of a three-dimensionalpoint. Once the desired point is located, its accuracy and precision areenhanced by detecting and processing the region of interest in theimage(s) captured by the camera using computer vision techniques andDeep Learning.

Through the information obtained by Ultrasound sensors or othercomponents, which are fixedly distributed in the environment (e.g.,mounted on an airplane fuselage), the space in which the user is locatedis reconstructed or reconstituted. The user also has an ultrasoundsensor or emitted that is used to triangulate between the user's pointof location and the fixed sensors and/or detectors in the environment.

The example methodology starts with the choice of a physical referencethat will serve as the origin. This reference is used to calibrate thevirtual coordinate systems of the ultrasound and the 3D reconstruction.After calibrating the respective virtual coordinate systems of theultrasound sensor system and the virtual 3D environment for the sameorigin (e.g., by transforming the virtual environment to world space asdefined by the ultrasound coordinate system), registration of sites ofinterest by the user, such as sites that need to be monitored in futureinspections, called “events,” are also registered. The informationrecorded are the spatial coordinates of the chosen sites and photos ofthese sites, as shown by FIG. 3-A similar to geomapping commonly usedfor online mapping of GPS locations such as tourist attractions.

With this information in the system, a different user (e.g., themechanic) (or the same user at a different, later time) can open theapplication interface, which will display visual information, such as anarrow through Augmented Reality, to guide the user to the spatiallocation of the event that was recorded in the prior step (see FIG.3-B). During the user's path to the marked event (from FIG. 3-A), theuser's position (pose) is determined with high precision. Computationalvision techniques associated with Deep Learning algorithms are used toenhance the positioning accuracy, so that the application interface canshow to the user exactly where the registered event by the initial user(see FIG. 3-C) is located.

The aim of the computer vision techniques and Deep Learning algorithmsis to capture images generated by an input sensor (e.g., a camera) inorder to segment and precisely recognize the area of interest, to make acomparison with the images that were registered in the system. Anexample non-limiting flowchart of this methodology is shown below inFIG. 4.

As mentioned above, for the operation of one proposed no-limitingsystem, Ultrasound sensors or other components are installed in theenvironment of interest (502). This ultrasound system is based on thepropagation of ultrasonic frequency sound through the air using, forexample, a piezoelectric device(s) to generate sound pulses (typicallyabove the range of human hearing, e.g., at 20 KHz or above), allowingthe sensors or other components to communicate with each other. Thesesensors or other components, called “anchors”, are fixed in places inthe environment and are configured in a mesh topology which can activelytrack a mobile sensor and/or emitter held or worn by the user. This setof sensors or other components uses electrical energy to performtransmission and signal reception, and to enable triangulation of theuser's spatial positioning within the anchor sensor mesh. Generallyspeaking, a single emitter-sensor pair allows detection of distance, twoemitter-sensor pairs (e.g., an emitter and two sensors or a sensor andtwo emitters) enables determination of a two-dimensional distancevector, and three emitter-sensor pairs enables detection of positioncoordinates in three dimensions. Additional enhancements (e.g., twosensors mounted close to one another on a handheld device) may be usedto detect aspects of orientation to enable sensing of pose in 4 degreesof freedom.

The 3D reconstruction may use cameras, infrared sensors and/or depthsensors (e.g., RADAR and/or LIDAR, or systems such as Microsoft's Kinect3D sensor) to reconstruct virtually the real environment and thusidentify the user's spatial position (e.g., as detected by theultrasonic system) in relation to the virtual environment (504). Theaccuracy of the user's position is improved by combining the informationobtained by the 3D cameras and the ultrasonic sensor system. In order tofurther ensure a high precision in the location of an event, techniquesof computer vision associated with algorithms of Deep Learning (DL) areapplied.

Once the user's position is known, the Augmented Reality system displaysthe registered event's spatial coordinates and guides the user to thedesired event through arrows or other indicators in the mobile devicedisplay (and/or may deliver audible instructions to the user such as“walk 10 paces ahead, locate the access panel labeled “‘Do Not Step’ andturn the handle 90 degrees counterclockwise to release the access panelretaining mechanism”). This system and method for spatial location ofthree-dimensional points using the combination of these technologiesprovides better signal spatial coverage, which translates into at leastthe following advantages: a shorter time for component localization toperform the maintenance task, high precision in the location of theregistered events, and greater tolerance to mobile obstacles (e.g., thepeople traffic, cars and objects in general). Additionally, the use ofAugmented Reality can guide the user to the place of interestefficiently and accurately.

In the exemplary embodiment presented below, the expression “events” canmean structural failures (delamination, debonding, cracks, corrosion,etc.) of electrical/hydraulic equipment (avionics, connectors, cables,sensors, tubes, etc.). The present embodiment is exemplary andnon-limiting of other applications in the aircraft industry, avionics orother environments which for example direct the user to items ofinterest other than “events.”

FIG. 6 illustrates a non-limiting embodiment of an array of sensors orother ultrasonic devices, embedded in an avionics compartment 600,configured in a mesh topology, used to determine the location of adisplay device. In the system described herein, the sensors or otherdevices (a, b, c, d . . . ) are electronic components capable oftransmitting and/or receiving signals in order to determine the locationof the display device within an avionics compartment 600.

In one non-limiting embodiment, the embedded array of sensors (a, b, c,d . . . ) or other devices function as ultrasonic sensors configured tosense a signal emitted from a user-carried display device via an emittercomponent that is either part of the display device or a moduleattachment to the display device or a device the user wears. In thisexample, the display device emits ultrasonic sound that the system usesto determine the 3D coordinates of the location (pose) of the displaydevice within the avionics compartment 600.

The emitter of the display device and the array of ultrasonic sensorsare operationally coupled to a computing system of the aircraft. Thecomputing system controls when the emitter emits an ultrasonic pulseand/or is notified when the emitter emits a pulse. The computing device(or hardware operatively connected to the computing device) times howlong it takes for the emitted signal to reach each sensor of an array ofultrasonic sensors. The computing device uses this timing information tocalculate the location of the display device. In one embodiment, theemitter is part of the display device of a user, and the anchors are thearray of ultrasonic sensors embedded at known locations on the aircraftfuselage.

Another preferred non-limiting embodiment, the display device isequipped with a sensor or other ultrasonic receiver component that iseither part of the display device or a module attachment to the displaydevice. This receiver is configured to sense signals emitted by an arrayof emitters embedded in the avionics compartment 600 of the aircraft.The emitters can be controlled to emit pulses in a known sequence and/orusing conventional signal marking techniques (so the sensor candistinguish the ultrasonic pulses emitted by the various emitters andmatch up a received pulse with a known emitter location).

In yet another non-limiting embodiment, the display device and/or theanchors of the aircraft are electronic components with transceivingproperties. This embodiment is configured so a transceiver emits a pulsewhich bounces/reflects off of a target and is received by the same ordifferent transceiver anchored onto the avionic compartment 600 or heldby the user. Thus, some embodiments can have an active device(s)anchored only within the environment, other embodiments can have anactive device(s) mounted on or carried by the user, and still otherembodiments can have active devices both in the environment and on theuser.

All previously described non-limiting embodiments may be supplementedwith an image sensor, that is either part of the display device or amodule attachment to the display device, wherein a user determines aphysical reference. Furthermore, deep learning image processingtechniques are used to determine the location of the display device inthe aircraft more accurately in addition to the previously describedembodiments. Such deep learning neural networks can be trained using asequence of known images to for example recognize the features of theenvironment such as the fuselage of particular aircraft.

Example Use Case

In this example of non-limiting embodiments, the proposed method andsystem is used for the spatial location of the three-dimensionallocation of an electrical connector (reference name: P0813) that is inthe aircraft avionics compartment (see FIG. 5-A).

Consider a hypothetical situation described below:

-   -   a) The aircraft monitoring system informs a fault message;    -   b) The fault message has an associated troubleshooting        procedure;    -   c) The troubleshooting procedure (see FIG. 5-B) requests a check        of an electrical connector (P0813) to correct this failure. This        electrical connector is located in the aircraft avionics        compartment;    -   d) The non-limiting technology guides the mechanic to the        requested event so that the mechanic can quickly, with accuracy        and precision, find the electrical connector P0813 that is        located in the aircraft avionics compartment; this enables the        mechanic to perform the tasks required by the troubleshooting        procedure.

Any patents and publications cited above are hereby incorporated byreference.

While the non-limiting technology has been described in connection withwhat is presently considered to be the most practical and preferredembodiments, it is to be understood that the invention is not to belimited to the disclosed embodiments, but on the contrary, is intendedto cover various modifications and equivalent arrangements includedwithin the spirit and scope of the appended claims.

The invention claimed is:
 1. A method of locating an event usingaugmented-reality, comprising; a. using a network of ultrasonictransducers, emitting a signal into an environment including anaircraft; b. using the network of ultrasonic transducers, detecting theemitted signal; c. processing time of flight of the signals between thetransducers to determine spatial location of a display device in theenvironment by triangulation; d. using deep learning to detect aphysical reference point in the environment; and e. displaying, on thedisplay device in response to the determined display device location andthe detected physical reference point, augmented-reality content forguiding the display device to an event on or in the aircraft.
 2. Themethod of claim 1, further including using the detected physicalreference point to calibrate with a coordinate system of theenvironment.
 3. The method of claim 1, further including registering aspatial coordinate of the event and guiding the display device to theregistered event based on the spatial coordinate.
 4. The method of claim1, further including tracking the display device by determining time offlight between at least one ultrasonic emitter and at least oneultrasonic sensor.
 5. The method of claim 4, wherein the at least oneultrasonic emitter or the at least one ultrasonic sensor is disposed onthe display device.
 6. The method of claim 4, further includingdetermining pose of the display device.
 7. The method of claim 4,wherein the at least one emitter or the at least one sensor comprises amesh topology.
 8. The method of claim 1, wherein the generatedaugmented-reality content, displayed by the display device, comprisesarrows, text boxes, virtual thermography, and other figures for pointingto the event.
 9. The method claim 7, wherein the generatedaugmented-reality content displayed on the display device providesinstructions or other maintenance information.
 10. An aircraft systemconfigured to locate an event using mixed-reality content, the systemcomprising: a. a network of ultrasonic devices, at least some of whichare embedded in an aircraft, configured to determine the dynamiclocation of a display device as it moves with respect to an environmentcontaining an aircraft; b. an image sensor configured to capture animage of the environment; and c. a processor coupled to the imagesensor, the processor using the captured image and time of flight ofsignals between the ultrasonic transducers to determine a referenceposition in the environment and to use the determined reference positionand the determined dynamic location to generate a mixed reality imagefor display on the display device, the mixed reality image including anindication of at least one part of the aircraft to be maintained. 11.The aircraft system of claim 10, wherein the processor calibrates acoordinate system using the determined reference position and thedetermined dynamic location.
 12. The aircraft system of claim 11,wherein the processor is configured to implement a deep learning neuralnetwork to determine the reference position.
 13. The aircraft system ofclaim 10, wherein the processor is coupled to the network of ultrasonicdevices and tracks the display device by measuring the changing time ofarrival (TOA) of ultrasonic signals exchanged between the ultrasonicdevices.
 14. The aircraft system of claim 10, wherein the display deviceincludes at least one of the ultrasonic devices.
 15. The aircraft systemof claim 10, wherein the mixed reality image comprises arrows, textboxes, virtual thermography, and other figures for pointing to an event.16. The aircraft system of claim 15, wherein the mixed reality imagecomprises instructions for maintaining the aircraft.
 17. A system forlocating an event using augmented-reality content, comprising; a. anetwork of ultrasonic devices, capable of receiving and transmittingultrasonic signals, configured to determine spatial coordinates bymeasuring time of arrival (TOA) of ultrasonic pulses between theultrasonic devices; b. a further sensor configured to measure athree-dimensional characteristic of an environment including anaircraft; and c. at least one processor coupled to the network ofultrasonic devices and the further sensor, the at least one processorbeing configured to use the determined spatial coordinates and themeasured three-dimensional characteristic to position virtual contentwithin a 3D model of the environment.
 18. The system of claim 17 whereinthe at least one processor is further configured to register an image ofan event with respect to the 3D model.
 19. The system of claim 17wherein the at least one processor is further configured to use a deeplearning neural network to measure the three-dimensional characteristic.20. The system of claim 17 further including a mobile display device,the at least one processor being further configured to generateaugmented-reality display content on the mobile display device.